Mathematics Education Under the Gavel: Who Controls the Minimum Competency Standards?

In 2019, the Ontario Ministry of Education announced a mandatory mathematics examination for all newly licensed teachers in the province. The following winter, after a brief pilot and a few months during COVID of testing, a court case declared the mathematics examination unconstitutional, and it has been paused since January 2021. This paper discusses the research-based evidence that has led to support from a mathematics standpoint for such an examination, as well as the court case that has changed the ability of the province to provide licensing guidelines. We provide our research conclusions on why such an examination might be needed and cautions for considering the reasons that led to the court overturning the proficiency test.

THE ME CURRICULUM: HIGH-STAKES TESTING AND THE AMERICAN COVID-19 RESPONSE

American culture's emphasis on individualism has been identified by those directly engaged with fighting the COVID-19 pandemic and by popular cultural commentators as problematic in mustering the col-lective social consciousness and self-sacrifice necessary to mount a successful pandemic response. This concern is supported by emerging academic research which postulates a relationship between a culture's relative balance between individualism and collectivism and the success of its pandemic response. Big data analysis suggests an inverse relationship between individualistic cultural traits and success in pandemic mitigation. Although the relative advantages and disadvan-tages of American individualism have been debated since Tocqueville, a recent rise in the dominance of individualism within the American psyche has been noted by both sides of the political spectrum with conservative commentary expressing concern over radical individualism and more left-leaning commentary finding increasing acceptance of libertarian principles perilous to the common good. Paralleling this rise in individualism are neo-liberal efforts at education reform that culminated in No Child Left Behind and its successor Every Student Succeeds. Such comprehensive neo-liberal reform efforts engender in-creased cultural individualism in three ways: by narrowing curriculum definitions and content to a core that excludes issues of citizenship;by increasing direct instruction that limits students' opportunities to engage with others in meaningful educational tasks;and most impor-tantly, by creating an accountability superstructure based solely on de-contextualized evaluations of isolated students, educators, and schools. The decades-long dominance of neo-liberal reform efforts create an educational environment which reinforces the cultural individualism hampering a successful American response to COVID-19. © 2022, University of North Carolina at Greensboro. All rights reserved.

The Show Must Go On: Lessons Learned from Using Remote Proctoring in a High-Stakes Medical Licensing Exam Program in Response to Severe Disruption

The COVID-19 pandemic severely disrupted assessment models that were commonplace in the testing industry for decades. As a response to this disturbance, remote proctoring has emerged as a promising and potentially sound alternative to offer examinations, while adhering to public health authority guidelines. However, validity evidence in support of such a delivery modality with high-stake examinations, in comparison with traditional in-person testing, is still limited. The purpose of this study was to evaluate, compare and share lessons learned from the delivery of a high-stake medical licensing examination that was offered under two proctoring modalities. Over a few months, a total of 2070 candidates (46%) completed the exam in test centers, while 2475 (54%) completed it remotely. The analysis of technical difficulties suggests that the candidate experience was different across the two conditions. However, findings pertaining to candidate performance were encouraging and suggest that these discrepancies did not impact exam or examinee characteristics in any meaningful fashion. We believe that our findings therefore provide some support for the use of remote proctoring as a defensible alternative for completing a high-stakes examination.

Comparability and Integrity of Online Remote vs. Onsite Proctored Credentialing Exams

Since the onset of the pandemic in 2020, many credentialing organizations have incorporated online remote administration of their examinations to enable continuity of their programs. This paper describes a research study examining several high stakes credentialing examination programs that utilized mixed delivery modes, including online remote testing at home, as well as testing in test centers. Candidates were monitored in real time by a test proctor, either remotely by video camera, or in person. The study examined the comparability of test scores, instances of irregular candidate testing behavior (potential cheating), and candidate test taking experience ratings across modalities. Overall, results of the study indicated that test scores were psychometrically sound and comparable across modes, rates of suspect test taking behavior were low and not significantly different across modes, and candidate experience ratings were favorable and unrelated to testing modality. Implications for future practice and research are discussed.

Preparing Middle Grades Candidates for edTPA in Uncertain Times

Teacher candidates in North Carolina must earn a passing score on the edTPA assessment to get certified. The middle grades education program at Western Carolina University integrates aspects of the edTPA assessment throughout pre-student teaching coursework and field experiences to prepare candidates for this high-stakes assessment. Some of the edTPA practice assignments serve as key assessments that help the middle grades program faculty evaluate the program and make decisions about curriculum. The pivot to remote and blended learning formats on campus and in partner middle level schools affected the implementation of the edTPA-related assignments. The authors share some of the challenges of implementing edTPA practice portfolios during the pandemic as well as insights gleaned from their assessment of the data.

A Quantitative Evaluation of a Live Remote Proctoring Pilot

The COVID-19 pandemic has witnessed a renewed interest in Live Remote Proctoring (LRP), not only as a test availability measure but as a necessity to maintain business continuity. Many certification organizations have correspondingly provided LRP as an option for candidates. This study describes a retrospective, observational pilot study evaluating modality effects for three high-stakes certification examination programs administered concurrently in test center and LRP conditions. Also reported are summaries of a post-examination survey assessing drivers for selection of testing condition and candidate satisfaction, particularly with aspects of the LRP experience. Significant differences were observed in both distributions of test scores and test duration (both higher for test center candidates). Although explanatory variables are not completely understood, the authors offer insights as to factors influencing these outcomes. While satisfaction levels with LRP were reasonably positive, significant technical issues were reported by LRP candidates. The primary drivers for the selection of LRP were safety concerns related to the pandemic, simple convenience, and lack of test center availability.

Mathematics Education Under the Gavel: Who Controls the Minimum Competency Standards?

In 2019, the Ontario Ministry of Education announced a mandatory mathematics examination for all newly licensed teachers in the province. The following winter, after a brief pilot and a few months during COVID of testing, a court case declared the mathematics examination unconstitutional, and it has been paused since January 2021. This paper discusses the research-based evidence that has led to support from a mathematics standpoint for such an examination, as well as the court case that has changed the ability of the province to provide licensing guidelines. We provide our research conclusions on why such an examination might be needed and cautions for considering the reasons that led to the court overturning the proficiency test. © 2023, Ontario Institute for Studies in Education (OISE).

COVID-19 impact on high stakes assessment: a New Zealand journey of collaborative adaptation amidst disruption

New Zealand’s defined coastal boundaries, isolation and small population were favourable factors to minimise the spread of COVID-19. Decisive governmental leadership and a public willing to comply with high-level lockdown in the first phase, resulted in minimal disruption to assessment. But as the pandemic progressed through Delta and Omicron variants, concerns grew about equitable access to assessments, declining school attendance, and inequitable educational outcomes for students, especially of Māori and Pacific heritage. School and educational agency experiences of high stakes assessment in a period of uncertainty were examined through document analysis and research interviews. Using Gewirtz’s contextual analysis of the multi-dimensional and complex nature of justice, and Rogoff’s conceptual framework of three planes of socio-cultural analysis: the personal (learner), inter-personal (school) and institutional (educational agencies), revealed that though collaborative adaptations minimised assessment disruptions on wellbeing and equity of access, they did not transform high stakes assessment. [ FROM AUTHOR]

BACHELOR'S THESIS RATING AS AN ADMISSION CRITERION TO THE MASTER'S PROGRAM OF PSYCHOLOGY: A PSYCHOMETRICAL CASE STUDY

The COVID-19 pandemic influenced admissions testing for the master's degree program of psychology at Masaryk University in 2020. The administration of the standard paper-and-pencil knowledge test was not possible;therefore, we chose the bachelor's thesis ratings instead. This paper is a psychometrical case study that covers the development of criteria related to content validity, design selection, and results. Two randomly selected raters evaluated each thesis, and we equated their severity using a linear logistic test model (LLTM) under the item response theory (IRT) paradigm. This procedure resulted in unidimensional and unbiased scores equated across 18 judges and 2 terms (n1 = 82, n2 = 48). The reliability was comparable to the standard tests, rxx'= 0.869, and judge severity and criteria difficulty did not differ across them. The resulting ratings seem to be valid and no less fair than the written exam. The proposed method can serve other departments and other goals, not only as an entrance test. We share an analytical script and all the necessary materials to enhance the use of this method. © 2022 Masaryk University, Faculty of Arts. All rights reserved.

Appraisal of high-stake examinations during SARS-CoV-2 emergency with responsible and transparent AI: Evidence of fair and detrimental assessment

In situations like the coronavirus pandemic, colleges and universities are forced to limit their offline and regular academic activities. Extended postponement of high-stakes exams due to health risk hereby reduces productivity and progress in later years. Several countries decided to organize the exams online. Since many other countries with large education boards had an inadequate infrastructure and insufficient resources during the emergency, education policy experts considered a solution to simultaneously protect public health and fully resume high-stakes exams -by canceling offline exam and introducing a uniform assessment process to be followed across the states and education boards. This research proposes a novel system using an AI model to accomplish the complex task of evaluating all students across education boards with maximum level of fairness and analyzes the ability to fairly appraise exam grades in the context of high-stakes examinations during SARS-CoV-2 emergency. Basically, a logistic regression classifier on top of a deep neural network is used to output predictions that are as fair as possible for all learners. The predictions of the proposed grade-awarding system are explained by the SHAP (SHapley Additive exPlanations) framework. SHAP allowed to identify the features of the students' portfolios that contributed most to the predicted grades. In the setting of an empirical analysis in one of the largest education systems in the Global South, 81.85% of learners were assigned fair scores while 3.12% of the scores were significantly smaller than the actual grades, which would have had a detrimental effect if it had been applied for real. Furthermore, SHAP allows policy-makers to debug the predictive model by identifying and measuring the importance of the factors involved in the model's final decision and removing those features that should not play a role in the model's “reasoning” process.

Commentary: Comment on college admissions tests and social responsibility

Comments on an article by M. Koljatic et al. (see record 2021-40254-001). COVID-19 pandemic disrupted the process of teaching and learning at every conceivable level, it had been hard to ignore a growing sentiment that standardized testing seems to do more to worsen inequality of opportunities than it does to expose it. Although the validity of this sentiment is a matter for debate, if ever there was an occasion for the individuals and organizations involved in the national and international testing industry to engage in some introspection about the value proposition of large-scale assessment across a variety of use contexts, that time would seem to be now. As the 2021-2022 president of the National Council on Measurement in Education (NCME), this is something and find myself pondering quite seriously. (PsycInfo Database Record (c) 2022 APA, all rights reserved)

"Designing into the Unknown": Harnessing the Promise of Flexible Course Design

With the return to on-campus learning in progress, recalling how higher education responded to the COVID-19 pandemic poses a challenge. Back in March 2020, many university leaders thought returning "to normal" (now a phrase almost as maligned as "unprecedented") was a matter of weeks--not months or years. The short-sighted certainty of the initial response as a brief pivot soon morphed into a mixture of horror and uncertainty about how to respond to the larger public health crisis and its consequences for higher education. Collaborating at scale with 200 colleagues and via an iterative learning community conversations, the authors confronted anxieties, frustrations, and new ways of seeing course design and inclusive teaching and learning. The authors shared what this immersive process brought forth at a January 2021 Association of American Colleges and Universities (AAC&U) panel entitled "Threshold Concepts in Online and Hybrid Course Design" in the form of three insights about flexible course design: community is co-created with, between, and centered on the whole student;effective assessment is an ongoing conversation;and good teaching requires continuous reflection and recalibration to authentically enact values. Following a brief description of the summer 2020 workshops, this article elaborates on these insights and considers their implications for teaching and educational development in a postpandemic future.

What is at stake without high-stakes exams? Students' evaluation and admission to college at the time of COVID-19.

The outbreak of COVID-19 in 2020 inhibited face-to-face education and constrained exam taking. In many countries worldwide, high-stakes exams happening at the end of the school year determine college admissions. This paper investigates the impact of using historical data of school and high-stakes exams results to train a model to predict high-stakes exams given the available data in the Spring. The most transparent and accurate model turns out to be a linear regression model with high school GPA as the main predictor. Further analysis of the predictions reflect how high-stakes exams relate to GPA in high school for different subgroups in the population. Predicted scores slightly advantage females and low SES individuals, who perform relatively worse in high-stakes exams than in high school. Our preferred model accounts for about 50% of the out-of-sample variation in the high-stakes exam. On average, the student rank using predicted scores differs from the actual rank by almost 17 percentiles. This suggests that either high-stakes exams capture individual skills that are not measured by high school grades or that high-stakes exams are a noisy measure of the same skill.

Exams tested by Covid-19: An opportunity to rethink standardized senior secondary examinations.

The global Covid-19 pandemic is testing the responsiveness of school systems. Extensive discourse about disruptions to the standardized examinations students take in their final year of secondary school is symbolic of their high-stakes status worldwide. The interruptions provide an opportune moment to question the efficacy of exams as a measurement of achievement. To explore these issues, this article shares some on-the-ground illustrations from Australian teachers about how high-stakes exams shape their enactment of senior secondary history curriculum. The presence of a discourse of exam alignment, which places a disproportionate emphasis on preparing students for exams, has implications for teachers' curricular practices and wider equity issues. These issues resonate in other international settings, especially during the pandemic. The severity of the Covid-19 economic downturn means it is more important than ever to investigate the relationship between curricular practices and socio-economic structures, to ensure examinations do not compound educational disadvantage.